In [1]:
from IPython.display import Image
Image(url='https://img4.cityrealty.com/neo/i/p/mig/airbnb_guide.jpg')
Out[1]:
AirBnB Data Analysis with Python Final Project
Content:
We are focusing on California’s data for analysis. Being new to California, it would be interesting to learn about places which people prefer for AirBnb in California. Our main focus was around affordability, customer preference for AirBnb property based on customer reviews and how close are these AirBnb properties from airport, beach and other touristy location.
Dataset:
https://www.kaggle.com/gkdoad/airbnb
Team:
- Resham Uttamchandani - ruttamchandani@scu.edu
- Mukesh Ganesh - mganesh2@scu.edu
- Pratiksha Raval - praval@scu.edu
- Dnyanai Surkutwar - dsurkutwar@scu.edu
Following are a few questions that we aim to answer through our analysis:
- Affordability analysis: How do prices of listings vary by location?
- Airbnb property preference based on cutomer reviews.
- What are the different types of properties in and around LA? Do they vary by neighborhood?
- What neighborhoods are rated highly by guests?
- Customer preference based Airbnb recommendations.
Importing libraries
In [2]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import folium
import json
import branca
pd.set_option('display.max_columns',50)
pd.options.display.max_rows
Out[2]:
Setting the map location co-ordinates for the map graphs later
In [3]:
# Setting a base map
lat = 34.1000
long = -118.4662
Read the listings csv file and setting the index column as id
In [4]:
initFile = pd.read_csv('listings.csv',index_col= 'id',low_memory=False)
initFile.head(1)
Out[4]:
Description of Data
- The dataset comprises of three main tables:
- listings - Detailed listings data showing 106 atttributes for each of the listings. Some of the attributes used in the analysis are price, longitude, latitude, listing_type (categorical),neighbourhood_cleansed(categorical), ratings among others.
- reviews - Detailed reviews given by the guests with 6 attributes. Key attributes include review_scores_rating, review_scores_accuracy, review_scores_cleanliness, review_scores_checkin, review_scores_communication, review_scores_location, review_scores_value, cancellation_policy.
- A quick glance at the data shows that there are:</font>
- Most of the listings in California are based in and near LA, predominately in LA county.
- Over 35,959 reviews have been written by guests since 26th May 2009.
- The price for a listing ranges from 10 per night to 25,000 dollars(!) per night. Listing with $25,000 price tag are in Hollywood hills west.
In [5]:
initFile.first_review.count()
Out[5]:
In [6]:
initFile.first_review.sort_values(ascending=True).head(1)
Out[6]:
In [7]:
initFile.head()
Out[7]:
Reseting the index and assigning a new column as the dataset id
In [8]:
initFile.reset_index(inplace=True)
In [9]:
initFile['colid'] = initFile.index
In [10]:
initFile.drop('id',axis=1,inplace=True)
In [11]:
initFile.rename(columns={'colid':'id'},inplace=True)
initFile.id +=1
In [12]:
initFile.set_index('id',inplace=True)
initFile.head(1)
Out[12]:
Data analysis and data Cleaning:¶
Cleaning the dataset to make sure we have only CA related values
In [13]:
df_cleansed = initFile.copy()
In [14]:
### Taking only the CA listings and making sure everything in state as uppercase
df_cleansed.state.loc[df_cleansed.state=='California'] = 'CA'
df_cleansed.state.unique()
df_cleansed.state = df_cleansed['state'].str.upper()
In [15]:
## As we can see there are some unwanted & non-CA entries in the state column
df_cleansed.state.value_counts()
Out[15]:
We need to remove the non-CA values from the dataset
In [16]:
df_cleansed.drop(df_cleansed[~(df_cleansed.state.values=='CA')].index,axis=0,inplace=True)
Convert the city column to lower case
In [17]:
df_cleansed.city = df_cleansed.city.str.lower()
df_cleansed.city.unique()
df_cleansed.city.dropna(inplace=True)
Cleaning the city column and dropping NaN values in city column
In [18]:
df_cleansed['city'] = df_cleansed['city'].str.replace(r'[^\x00-\x7F]+', '')
df_cleansed.city.dropna(inplace=True)
df_cleansed.city.value_counts()
Out[18]:
In [19]:
df_cleansed.groupby('city')['neighbourhood_cleansed'].value_counts()
Out[19]:
In [20]:
df_cleansed.smart_location.shape
Out[20]:
In [21]:
df_cleansed.smart_location = df_cleansed.smart_location.str.lower()
In [22]:
df_cleansed.isnull().sum()
Out[22]:
Cleaning the smart_location column, removing the non-English characters in the smart_location column
In [23]:
df_cleansed.drop(df_cleansed[(df_cleansed['smart_location'] == '蒙特利帕克, ca') | \
(df_cleansed['smart_location'] == '因達斯特里, ca') | \
(df_cleansed['smart_location'] == '艾尔蒙地, ca') | \
(df_cleansed['smart_location'] == '沃尔纳特, ca') | \
(df_cleansed['smart_location'] == '哈仙达岗, ca') | \
(df_cleansed['smart_location'] == '罗兰高地, ca') | \
(df_cleansed['smart_location'] == '罗兰岗, ca') | \
(df_cleansed['smart_location'] == '天普市, ca') | \
(df_cleansed['smart_location'] == '洛杉矶, ca') \
].index, inplace = True)
Replacing ', ca' in smart_location column so that it is uniform with the other location related columns
In [24]:
df_cleansed.smart_location = df_cleansed.smart_location.str.replace(', ca',' ', regex=True)
df_cleansed.smart_location = df_cleansed.smart_location.str.strip()
## Removing the spaces from the neighbourhood_cleansed
df_cleansed.neighbourhood_cleansed = df_cleansed.neighbourhood_cleansed.str.strip()
In [25]:
## Convert smart_location and neighbourhood_cleansed columns into lower case
df_cleansed.smart_location = df_cleansed.smart_location.str.lower()
df_cleansed.neighbourhood_cleansed = df_cleansed.neighbourhood_cleansed.str.lower()
df_cleansed.head(2)
Out[25]:
Interesting Fact 1¶
Q) Make a dataset copy for Interesting Fact 1 which is focused on:
Affordability analysis: How do prices of listings vary by location
Fact 1 insights:
For the Low price range,almost all of the prices in Low price range are located in Hollywood.
Other top locations with maximum low price range are: Venice, pico-union, Downtown and Long Beach.
The average price in the Low range is around 100 dollars.
For the Mid price range,almost all of the prices in the Mid price range are located in Venice.
Other top locations with maximum mid price range are: Hollywood, Downtown and Long Beach.
The average price in the Mid range is around 150-200 dollars.
As observed above,Almost all of the prices in the High price range are located in Venice and Beverly grove.Venice seems a real versatile location for Airbnb listings.
Other top locations with maximum high price range are: Hollywood hills West, Malibu and Avalon.
The average price in the High range is around $550-800.
Other top locations with maximum low price range are: Venice, pico-union, Downtown and Long Beach.
The average price in the Low range is around 100 dollars.
Other top locations with maximum mid price range are: Hollywood, Downtown and Long Beach.
The average price in the Mid range is around 150-200 dollars.
Other top locations with maximum high price range are: Hollywood hills West, Malibu and Avalon.
The average price in the High range is around $550-800.
In [26]:
df_Fact1 = df_cleansed[['price','security_deposit','cleaning_fee','guests_included','extra_people','minimum_nights',\
'maximum_nights','amenities','neighbourhood_cleansed',\
'state','zipcode','latitude','longitude']].copy()
## Renaming the neighbourhood_cleansed column to location
df_Fact1.rename(columns={'neighbourhood_cleansed':'location'},inplace=True)
In [27]:
df_Fact1.info()
In [28]:
df_Fact1.isna().sum()
## We see that we have NaN values in the security_deposit, cleaning_fee, zipcode columns
## So, lets clean these columns
Out[28]:
In [29]:
## Lets see what the amenities column holds..
df_Fact1.groupby('amenities')['price'].value_counts().nlargest(3)
Out[29]:
Cleaning the Amenities column and converting it to lower case
In [30]:
df_Fact1.amenities = df_Fact1.amenities.str.strip('{}')
df_Fact1.amenities = df_Fact1.amenities.str.replace('"','')
df_Fact1.amenities = df_Fact1.amenities.str.lower()
Trying to understand which are the more frequently used values in the amenities column
In [31]:
pd.Series(' '.join(df_Fact1['amenities']).lower().split(',')).value_counts().nlargest(12)
Out[31]:
Creating new coulmns around most frequently used values in the Amenities column
In [32]:
## Taking the most sort after amenities by customers and creating new columns for them
df_Fact1['wifi'] = ''
df_Fact1['ac'] = ''
df_Fact1['pet'] = ''
df_Fact1['park'] = ''
df_Fact1['htub'] = ''
df_Fact1['dishwasher'] = ''
df_Fact1['bbq'] = ''
df_Fact1['pool'] = ''
df_Fact1['hr'] = ''
df_Fact1['coffee'] = ''
df_Fact1['heating'] = ''
df_Fact1['iron'] = ''
df_Fact1['washer'] = ''
df_Fact1['dryer'] = ''
df_Fact1['parking'] = ''
If the amenities satisfy the condition of being present in the Amenities column then set the column as 'True'
In [33]:
df_Fact1.wifi = df_Fact1.amenities.str.contains("wifi").apply(lambda x: 'wifi' if x==True else '')
df_Fact1.ac = df_Fact1.amenities.str.contains("air conditioning").apply(lambda x: 'air conditioning' if x==True else '')
df_Fact1.pet = df_Fact1.amenities.str.contains("pet").apply(lambda x: 'pet' if x==True else '')
df_Fact1.park = df_Fact1.amenities.str.contains("park").apply(lambda x: 'park' if x==True else '')
df_Fact1.htub = df_Fact1.amenities.str.contains("hot tub").apply(lambda x: 'hot tub' if x==True else '')
df_Fact1.dishwasher = df_Fact1.amenities.str.contains("dishwasher").apply(lambda x: 'dishwasher' if x==True else '')
df_Fact1.bbq = df_Fact1.amenities.str.contains("bbq").apply(lambda x: 'BBQ' if x==True else '')
df_Fact1.pool = df_Fact1.amenities.str.contains("pool").apply(lambda x: 'pool' if x==True else '')
df_Fact1.hr = df_Fact1.amenities.str.contains("24-hour check-in").apply(lambda x: '24-hour check-in' if x==True else '')
df_Fact1.coffee = df_Fact1.amenities.str.contains("coffee").apply(lambda x: 'coffee' if x==True else '')
df_Fact1.heating = df_Fact1.amenities.str.contains("heating").apply(lambda x: 'heating' if x==True else '')
df_Fact1.iron = df_Fact1.amenities.str.contains("iron").apply(lambda x: 'iron' if x==True else '')
df_Fact1.washer = df_Fact1.amenities.str.contains("washer").apply(lambda x: 'washer' if x==True else '')
df_Fact1.dryer = df_Fact1.amenities.str.contains("dryer").apply(lambda x: 'dryer' if x==True else '')
df_Fact1.parking = df_Fact1.amenities.str.contains("parking").apply(lambda x: 'parking' if x==True else '')
Drop the old amenities column as we have made columns for amenities and did one-hot encoding
In [34]:
df_Fact1.drop('amenities',axis=1,inplace=True)
Adding amenities to the Amenities column
In [35]:
df_Fact1['amenities'] = df_Fact1.wifi+' '+df_Fact1.ac+' '+df_Fact1.pet+' '+df_Fact1.park+' '+\
df_Fact1.htub+' '+df_Fact1.dishwasher+' '+df_Fact1.bbq+' '+df_Fact1.pool+' '+df_Fact1.hr+' '+df_Fact1.coffee+' '+\
df_Fact1.heating+' '+df_Fact1.iron+' '+df_Fact1.washer+' '+df_Fact1.dryer+' '+df_Fact1.parking
Converting the individual amenities columns using one-hot encoding
In [36]:
df_Fact1.wifi = df_Fact1.wifi.apply(lambda x: 1 if x=='wifi' else 0)
df_Fact1.ac = df_Fact1.ac.apply(lambda x: 1 if x=='air conditioning' else 0)
df_Fact1.pet = df_Fact1.pet.apply(lambda x: 1 if x=='pet' else 0)
df_Fact1.park = df_Fact1.park.apply(lambda x:1 if x=='park' else 0)
df_Fact1.htub = df_Fact1.htub.apply(lambda x:1 if x=='hot tub' else 0)
df_Fact1.dishwasher = df_Fact1.dishwasher.apply(lambda x:1 if x=='dishwasher' else 0)
df_Fact1.bbq = df_Fact1.bbq.apply(lambda x:1 if x=='BBQ' else 0)
df_Fact1.pool = df_Fact1.pool.apply(lambda x:1 if x=='pool' else 0)
df_Fact1.hr = df_Fact1.hr.apply(lambda x:1 if x=='24-hour check-in' else 0)
df_Fact1.coffee = df_Fact1.coffee.apply(lambda x:1 if x=='coffee' else 0)
df_Fact1.heating = df_Fact1.heating.apply(lambda x:1 if x=='heating' else 0)
df_Fact1.iron = df_Fact1.iron.apply(lambda x:1 if x=='iron' else 0)
df_Fact1.washer = df_Fact1.washer.apply(lambda x:1 if x=='washer' else 0)
df_Fact1.dryer = df_Fact1.dryer.apply(lambda x:1 if x=='dryer' else 0)
df_Fact1.parking = df_Fact1.parking.apply(lambda x:1 if x=='parking' else 0)
In [37]:
df_Fact1.isna().sum()
Out[37]:
Working on the security_deposit column and adjusting its NaN values
In [38]:
## Removing the '$' & ',' symbols form the security_deposit column
df_Fact1.security_deposit = df_Fact1.security_deposit.str.strip('$')
df_Fact1.security_deposit = df_Fact1.security_deposit.str.replace(',','')
In [39]:
## Converting the security_deposit column to Float from str
df_Fact1.security_deposit = df_Fact1.security_deposit.astype('float64')
In [40]:
## Understanding the security_deposit column using describe()
df_Fact1.security_deposit.describe()
Out[40]:
In [41]:
## Rounding the security_deposit column values after filling the NaN values to mean() of
## the security_deposit values
df_Fact1.security_deposit.fillna(round(df_Fact1.security_deposit.mean(),2),inplace=True)
Working on the cleaning_fee column and adjusting its NaN values
In [42]:
## Understanding the cleaning_fee column using describe()
df_Fact1.cleaning_fee.describe()
Out[42]:
In [43]:
df_Fact1.cleaning_fee.isna().sum()
Out[43]:
In [44]:
## Removing the '$' & ',' symbols form the cleaning_fee column
df_Fact1.cleaning_fee = df_Fact1.cleaning_fee.str.strip('$')
df_Fact1.cleaning_fee = df_Fact1.cleaning_fee.str.replace(',','')
In [45]:
## Converting the cleaning_fee column to Float from str
df_Fact1.cleaning_fee = df_Fact1.cleaning_fee.astype('float64')
In [46]:
## Understanding the cleaning_fee column using describe()
df_Fact1.cleaning_fee.describe()
Out[46]:
In [47]:
## Converting the cleaning_fee column NaN values to the mean() of values in that column
df_Fact1.cleaning_fee.fillna(round(df_Fact1.cleaning_fee.mean(),2),inplace=True)
In [48]:
## Describing the cleaning_fee column after filling the NaN values in the column
df_Fact1.cleaning_fee.describe()
Out[48]:
In [49]:
df_Fact1.isna().sum()
Out[49]:
Checking the zipcode column and its NaN values
In [50]:
df_Fact1.zipcode.isna().sum()
Out[50]:
In [51]:
df_Fact1.zipcode.dropna(inplace=True)
In [52]:
df_Fact1.shape
Out[52]:
Checking the price column
In [53]:
## Removing the '$' & ',' symbols form the price column
df_Fact1.price = df_Fact1.price.str.strip('$')
df_Fact1.price = df_Fact1.price.str.replace(',','')
In [54]:
## Converting the price column to Float from str
df_Fact1.price = df_Fact1.price.astype('float64')
In [55]:
## Understanding the price column using describe()
df_Fact1.price.describe()
Out[55]:
In [56]:
## Price '0' makes no sense as the listing need to charge for per night stay and there are
## only 9 values
df_Fact1.drop(df_Fact1[df_Fact1.price==0].index,axis=0,inplace=True)
In [57]:
df_Fact1.price.count()
Out[57]:
Checking the extra_people column and making it numerical
In [58]:
## Removing the '$' & ',' symbols form the extra_people column
df_Fact1.extra_people = df_Fact1.extra_people.str.strip('$')
df_Fact1.extra_people = df_Fact1.extra_people.str.replace(',','')
In [59]:
## Converting the extra_people column to Float from str
df_Fact1.extra_people = df_Fact1.extra_people.astype('float64')
In [60]:
## Understanding the extra_people column using describe()
df_Fact1.extra_people.describe()
Out[60]:
In [61]:
df_Fact1.guests_included.dtype
df_Fact1.minimum_nights.dtype
df_Fact1.maximum_nights.dtype
## Since the numerical columns are alreay in integer type we do not need to make any change
Out[61]:
Dividing the price column into 10 equal parts
In [62]:
pd.qcut(df_Fact1.price,10)
Out[62]:
Create a new priceType column which will be have price ranges - Low,Mid,High
In [63]:
df_Fact1['priceType'] = ''
Dividing the price column into categories and assigning them a label in the priceType column
In [64]:
df_Fact1.priceType.loc[(df_Fact1['price']> 9.999) & (df_Fact1['price'] <= 110.0)] = 'Low'#Low
df_Fact1.priceType.loc[(df_Fact1['price'] > 110.0) & (df_Fact1['price'] <= 500.0)] = 'Mid'#Mid
df_Fact1.priceType.loc[(df_Fact1['price'] > 500.0) & (df_Fact1['price'] <= 25000.0)] = 'High'#High
We have created labels based on the Airbnb prices, lets have a look at the airbnb in these particular ranges
In [65]:
df_Fact1.head(1)
Out[65]:
First, we will check for the Low range Airbnb's
In [66]:
## Making a lowRange dataframe which has all the values with the priceType as "Low"
lowRnge = pd.DataFrame(df_Fact1[df_Fact1.priceType=='Low'])
lowRnge.head(2)
Out[66]:
Making the map for representing the Low ranged Airbnb listings using folium library
In [67]:
## Importing the markercluster module from the folium library
from folium.plugins import MarkerCluster
mc = MarkerCluster()
## Assigning base_low to save the map
### making the world map to zoom on the latitude and longitude of LA county as maximum
### number of airbnb listings are in LA county.
base_low = folium.Map(location=[lowRnge.latitude.mean(),
lowRnge.longitude.mean()],
zoom_start=10)##This simply zoom in on the CA state
## For every value in the Low range airbnb's put a popup marker with the Airbnb per night
## prices
for row in lowRnge.itertuples():
mc.add_child(folium.Marker(location=[row.latitude, row.longitude],popup=row.price))
base_low.add_child(mc)
## Saving the map in html format in the current directory with the name map_lowprices
base_low.save('map_lowprices.html')
Please note, you might have to run the jupyter notebook to see the map below or refer to the map_.html for map details
In [68]:
base_low
Out[68]: